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- The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH{S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

• Extensions of time may be available under the provisions of 37 CFR 1.136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 1 33). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
eamed patent term adjustment. See 37 CFR 1, 704(b). 

Status 

1 )S Responsive to communication(s) filed on 24 March 2004 . 
2a)n This action is FINAL. 2b)^ This action is non-final. 

3) 0 Since this application is in condition for allowance except for fomnal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 O.G. 213. 

Disposition of Claims 

4) ^ Claim(s) 8-14 and 17-30 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) n Clalm{s) is/are allowed. 

6) ^ Claimfs) 8-14 and 17-30 is/are rejected. 

7) 0 Claim(s) is/are objected to. 

8) 0 Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) n The specification Is objected to by the Examiner. 

10)0 The drawlng(s) filed on is/are: a)n accepted or b)^ objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 

Replacement drawing sheet(s) including the con-ection is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 
1 1 )□ The oath or declaration Is objected to by the Examiner. Note the attached Office Action or form PTO-152, 

Priority under 35 U.S.C. § 119 

12)0 Acknowledgment Is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 
a)n All b)n Some * c)^ None of: 

1 .□ Certified copies of the priority documents have been received. 

2. n Certified copies of the priority documents have been received in Application No. , 

3. n Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 
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Application/Control Number: 09/818,123 
Art Unit: 2672 



DETAILED ACTION 

1 . This office action is responsive to the following communications: 
Application, filed 03/27/2001; Amendment A, filed 06/30/2003; Amendment B, 
filed 12/05/2003; Amendment C, filed 03/24/2004. 

2. Claims 8-14 and 17-30 are pending in this application. Claims 8, 19, 23 
and 27 are independent claims. 

3. The present title of this application is "Augmented Reality Guided 
Instrument Positioning with Depth Determining Graphics" (as originally filed). 

Continued Examination Under 37 CFR 1.114 

4. A request for continued examination under 37 CFR 1.114, including the 
fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. 
Since this application is eligible for continued examination under 37 CFR 1.114, 
and the fee set forth in 37 CFR 1 .1 7(e) has been timely paid, the finality of the 
previous Office action has been withdrawn pursuant to 37 CFR 1.114. 
Applicant's submission filed on 03/24/2004 has been entered. 

Claim Rejections - 35 USC § 103 

5. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for 
all obviousness rejections set forth in this Office action; 

(a) A patent may not be obtained though the invention is not identically disclosed or described 
as set forth in section 102 of this title, if the differences between the subject matter sought to 
be patented and the prior art are such that the subject matter as a whole would have been 
obvious at the time the invention was made to a person having ordinary skill in the art to which 
said subject matter pertains. Patentability shall not be negatived by the manner in which the 
invention was made. 
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6. Claims 8-14 and 17-30 rejected under 35 U.S.C. 103(a) as being 
unpatentable over Fukunaga, U.S. Patent Number 6,346,940, "Virtualized 
Endoscope System", 02/12/2002, filed 02/27/1998, in view of Manwaring et al., 
U.S. Patent Number 5,638,819, "Method and Apparatus for Guiding an 
Instrument to a Target", 06/17/1997. 



Regarding claim 8, Fukunaga discloses an apparatus for augmented 
reality guided instrument positioning, comprising: a graphics guide generator 
(col. 7, line 65 - col. 8, line 10) for generating a first graphic target. 

Fukunaga further discloses (col. 2, lines 45-58, preparing a shape of an 
Image with a hidden surface) (col. 2, lines 64-65, a guiding marker preparation 
unit displaying guiding markers) (col. 2, lines 66-67, guiding markers used to 
guide the insertion of an actual or virtual endoscope) (col. 2, lines 59-63, a 
compass image preparation unit which generates a compass image over a first 
image) and further discloses in col. 2, lines 1-7, even Is the position is known the 
depths are difficult to discern and a user would have to estimate the depth of a 
cavity before advancing an endoscope tip. 

However, It is noted that Fukunaga fails to disclose depth marker 
indicating a first target depth of a hidden target, and a graphic instrument depth 
marker indicating a depth of an instrument; and a rendering module for rendering 
a virtual view of the first graphic target depth marker overlaid on an actual view of 
an object including the target, and rendering a virtual view of the graphic 
instrument depth marker overlaid on the actual view such that a proximity of a 
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predetermined portion of the instalment to the target is ascertainable based on a 
position of the graphic instrument marker with respect to the first graphic target 
depth marker. 

Manwaring discloses position data that informs the user of the instrument 
in relation to X,Y,Z coordinate system, Z being depth, of the distance between 
the instrument and a target, col. 5, lines 40-45, and knowing at any instant the 
location of the instrument to the alignment feature, col. 6, lines 23-35. 

Therefore, it would have been obvious to one of ordinary skill in the art at 
the time of the invention to include the depth information, as disclosed in 
Manwaring, of the instrument, alignment and target, as a marker, in the guiding 
marker preparation unit disclosed in Fukunaga, overlaid on the view of the target 
to ascertain the position of the instrument with respect to the target depth marker, 
to allow an operator to quickly and easily obtain guiding images for guiding an 
instrument through the actual cavity. 

Regarding claim 9, Fukunaga discloses graphics proximity marker 
generator determines an optimal location for the predetermined portion of the 
instmment with respect to the target and calculates the proximity of the 
predetermined portion of the instrument to the target based on the optimal 
location, (col. 2, lines 53-58, preparation unit displays a focus mark, which 
Examiner interprets as a graphics proximity marker, to mark a condition inclusive 
of the location of the condition in the object , because the mark is inclusive of the 
location, Examiner interprets the focus mark as being proximate to the 
predetermined portion of the instrument based on an optimal location) 
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Regarding claim 10, it is noted tliat Fukunaga fails to disclose the graphics 
guide generator detemnines a second graphic target depth marker indicating a 
second target depth of the target, rendered as a virtual view overlaid on the 
actual view by the rendering module, wherein a distance between the virtual view 
of the first graphic target depth marker and the virtual view of the second graphic 
target depth marker is a predetermined target depth range of the target. 

Manwaring discloses slices providing a reference to maintain a correct 
sense of orientation, col. 5, lines 14-16, and position data that informs the user of 
the instrument in relation to X,Y,Z coordinate system. Z being depth, of the 
distance between the instrument and a target, col. 5, lines 40-45, and knowing at 
any instant the location of the instrument to the alignment feature, col. 6, lines 
23-35. 

Therefore, it would have been obvious to one of ordinary skill in the art at 
the time of the invention to include second depth information, as disclosed in 
Manwaring, of the instrument, alignment and target, as a marker, in the guiding 
marker preparation unit disclosed in Fukunaga, overlaid on the view of the target 
to ascertain the position of each surface slice in the target, to allow an operator to 
quickly and easily obtain guiding images for guiding an instmment through the 
actual cavity. 

Regarding claim 1 1 , Fukunaga discloses the proximity corresponds to a 
final fonward position of the predetermined portion of the instrument with respect 
to the target, (col. 8, lines 1-10) 
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Regarding clainn 12, it is noted that Fukunaga fails to discloses graphics 
guide generator generates a third graphic target depth marker, rendered as a 
virtual view overlaid on the actual view by the rendering module indicating an 
outer surface of the target, wherein the virtual view of the first graphic target 
depth marker indicates an inner portion of the target. 

Manwaring discloses slices providing a reference to maintain a correct 
sense of orientation, col. 5, lines 14-16, and position data that informs the user of 
the instrument in relation to X,Y,Z coordinate system, Z being depth, of the 
distance between the instrument and a target, col. 5, lines 40-45, and knowing at 
any instant the location of the instrument to the alignment feature, col. 6, lines 
23-35. 

Therefore, it would have been obvious to one of ordinary skill in the art at 
the time of the invention to include second depth information, as disclosed in 
Manwaring, of the instrument, alignment and target, as a marker, in the guiding 
marker preparation unit disclosed in Fukunaga, overlaid on the view of the target 
to ascertain the position of each surface slice in the target, to allow an operator to 
quickly and easily obtain guiding images for guiding an instrument through the 
actual cavity. 

Regarding claim 13, proximity comprises a first measure of proximity for 
indicating a front portion of the target and a second measure of proximity for 
indicating a back portion of the target, the front portion corresponding to entering 
the target and the back portion corresponding to exiting the target, (col. 8, lines 
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1-10, markers indicate the path to take, which Examiner interprets as including a 
front to back and back to front proximity) 

Regarding claim 14, the graphic guide generator determines at least one 
graphics path marker that identifies at least one path for the instrument to the 
target; wherein said rendering module renders a virtual view of the at least one 
graphics path marker as an overlay on the actual view to which a portion of the 
instrument visible in the actual view is alignable. (col. 8, lines 1-10, markers 
indicate the path to take, and the type of guiding marker, and further discloses 
based on the viewpoint position and view direction the computation unit 
determines the position of the instrument for relative to an outer shape, view 
direction and reference direction, col. 13, lines 30-54, which Examiner interprets 
as an alignable feature) 

Regarding claim 17, a graphic guide generator for determining at least one 
graphics path marker that identifies a path to the target, wherein the graphics 
path marker is a shape comprising a window, wherein the window marks the 
path. (col. 8, lines 1-10, markers indicate the path to take, and the type of 
guiding marker) 

Regarding claim 18, a graphic guide generator for determining at least one 
graphics path marker that identifies a path to the target, wherein the graphics 
path marker is a pattern of lines centered on the target, wherein at least two lines 
are spaced differently from at least one other line, wherein the at least two lines 
frame the path. (col. 7, lines 65- col. 8, lines 10) 
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Regarding claims 19, 23 and 27, they are rejected based upon similar 
rational as above claim 8. 

Regarding claims 20, 24, and 28, they are rejected based upon similar 
rational as above claim 10. 

Regarding claims 21, 25, and 29, they are rejected based upon similar 
rational as above claim 10. 

Regarding claims 22, 26 and 30, they are rejected based upon similar 
rational as above claim 14. 

Any inquiry concerning this communication or earlier communications from 
the examiner should be directed to Motilewa A. Good-Johnson whose telephone 
number is (703) 305-3939. The examiner can normally be reached on Monday - 
Friday 8:30 AM - 5:00 PM. 

If attempts to reach the examiner by telephone are unsuccessful, the 
examiner's supervisor, Mike Razavi can be reached on (703) 305-4713. The fax 
phone number for the organization where this application or proceeding is 
assigned is 703-872-9306. 
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Information regarding the status of an application may be obtained from 
the Patent Application Information Retrieval (PAIR) system. Status information 
for published applications may be obtained from either Private PAIR or Public 
PAIR. Status information for unpublished applications is available through 
Private PAIR only. For more infonnation about the PAIR system, see http://pair- 
direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll- 
free). 
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